A Proposed Model for Simulation Validation Process Maturity

The software and systems communities have both advanced the state of their practices by applying models of capabilities to gauge the maturity of their practitioners and their processes [1, 2]. These models build capability maturity in layers that successively lead to greater end product quality. The success of these capability maturity models suggests that such a concept could promote the advancement of simulation validation practice as well. Others have considered this problem both for simulation in general [3] and simulation verification, validation, and accreditation (VV&A) [4]. Scholten and Udink ten Cate have developed a simulation maturity model (SMM) in which they show the role of validation in promoting simulation quality [3]. Conwell, Enright, and Stutzman demonstrate how to use the Carnegie Mellon University Software Engineering Institute’s (SEI’s) software and system acquisition capability maturity models (SW-CMM and SA-CMM) to improve simulation development and acquisition [4]. Both of these approaches demonstrate the importance of validation to the simulation development and use This paper proposes a model of process maturity for simulation validation. The development of this model begins by recognizing validation as a process that generates information as its sole product and therefore resembles a systematic quest for truth. These characteristics distinguish the simulation validation process from other processes such as those for manufacturing or software engineering. This development then substitutes process objectivity and the properties of quality information for the difficult-to-measure qualities of truth. The properties of information quality are defined by the completeness and correctness of the information and the confidence that the information producer has that the information is complete and correct enough to serve a particular purpose. These parameters come by amalgamating the notions for information quality held by the process improvement and traditional scientific communities. The validation process takes validation criteria, the referent, the simulation’s conceptual model, verification results of the intermediate development products, and the simulation results as input and produces evidence assessing the simulation’s validity as output. The quality of the input information ultimately limits the quality of the evidence that the validation process produces. The proposed model of validation process maturity thus structures the validation process into six levels defined by the quality of the input information, the quality of its information products, and the objectivity of those process components that contribute to those products. At the first level, the users demand absolutely no validation of the models they apply. The second level consists entirely of face validation where the validation evidence depends entirely upon subjective sources of requirements, referent, and validity judgments. The next level improves the objectivity and quality of the representational requirements. The next two levels progressively improve the objectivity of the conceptual modeling and results validation component processes and the quality of the evidence they produce. In the final maturity level, the validation process automatically transforms informal user need statements into formal validation criteria and then applies mathematical techniques to prove conceptual model and simulation results validity. In this model, the objectivity and quality of the validation evidence increases as the level of simulation validation process maturity increases.